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t-h 1 Abstract 

(N : 

Recently, [14,28] theoretically analyzed the success of a polynomial l\ -optimization algorithm in solv- 
ing an under-determined system of linear equations. In a large dimensional and statistical context [14,28] 
HH proved that if the number of equations (measurements in the compressed sensing terminology) in the system 

is proportional to the length of the unknown vector then there is a sparsity (number of non-zero elements of 
the unknown vector) also proportional to the length of the unknown vector such that l\ -optimization suc- 
ceeds in solving the system. In this paper, we provide an alternative performance analysis of l\ -optimization 
■ and obtain the proportionality constants that in certain cases match or improve on the best currently known 

\ ones from [28,29]. 

\o . 

\0 ■ 

Index Terms: Compressed sensing; t\ -optimization . 
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o : 
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O ■ 1 Introduction 

> : 

^ ■ In last several years the area of compressed sensing has been the subject of extensive research. The break- 

through results of [14] and [28] theoretically demonstrated that in certain applications (e.g. signal pro- 
cessing in sensor networks) classical sampling at Nyquist rate may not be necessary to perfectly recover 
signals. Namely, it turns out that a crucial compressed sensing problem is finding the sparsest solution of an 
under-determined system of equations. While this problem had been known for a long time it is the work 
of [14] and [28] that rigorously proved for the first time that a sparse enough solution can be recovered by 
solving a linear program in polynomial time. These results generated enormous amount of research with 
possible applications ranging from high-dimensional geometry, image reconstruction, single-pixel camera 
design, decoding of linear codes, channel estimation in wireless communications, to machine learning, data- 
streaming algorithms, DNA micro-arrays, magneto-encephalography etc. (more on the compressed sensing 
problems, their importance, and wide spectrum of different applications can be found in excellent refer- 
ences [4, 12, 15, 24, 36, 58, 61, 67, 69, 71 , 72, 91 , 93]). 
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In this paper we are interested in the mathematical background of certain compressed sensing problems. 
As is well known, these problems are very easy to pose and very difficult to solve. Namely, they are as 
simple as the following: we would like to find x such that 

Ax = y (1) 

where A is an m x n (m < n) measurement matrix and y is an m x 1 measurement vector. Standard 
compressed sensing context assumes that x is an n x 1 unknown /c-sparse vector (see Figure [T} here and 
in the rest of the paper, under /c-sparse vector we assume a vector that has at most k nonzero components). 
The main topic of this paper will be compressed sensing of the so-called ideally sparse signals (more on 
the so-called approximately sparse signals can be found in e.g. [21, 80, 84,95]). We will mostly throughout 
the paper assume no special structure on the sparse signal (more on the very relevant cases of sparse signals 
with special structures the interested reader can find in [4,6,9, 17, 18,23,38-41,44,55,56,62,65,66,79, 82, 
83, 85, 87, 89,91,96-98]). Also, in the rest of the paper we will assume the so-called linear regime, i.e. we 
will assume that k = fin and that the number of the measurements is m = an where a and fj are absolute 
constants independent of n (more on the non-linear regime, i.e. on the regime when m is larger than linearly 
proportional to k can be found in e.g. [22,45,46]). 
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Figure 1 : Model of a linear system; vector x is /c-sparse 



We generally distinguish two classes of possible algorithms that can be developed for solving (fl]). The 
first class of algorithms assumes freedom in designing the measurement matrix A. If one has the freedom 
to design the measurement matrix A then the results from [3, 59, 66] demonstrated that the techniques from 
coding theory (based on the coding/decoding of Reed-Solomon codes) can be employed to determine any 
/c-sparse x in (OQ) for any < a < 1 and any fi < f in polynomial time. It is relatively easy to show 
that under the unique recoverability assumption (5 can not be greater than tt. Therefore, as long as one is 
concerned with the unique recovery of /c-sparse x in £T|) in polynomial time the results from [3, 59, 66] are 
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optimal. The complexity of algorithms from [3,59,66] is roughly 0(n 3 ). In a similar fashion one can, 
instead of using coding/decoding techniques associated with Reed/Solomon codes, design the measurement 
matrix and the corresponding recovery algorithm based on the techniques related to the coding/decoding of 
Expander codes (see e.g. [52,53,94] and references therein). In that case recovering x in (Q} is significantly 
faster for large dimensions n. Namely, the complexity of the techniques from e.g. [52,53,94] (or their slight 
modifications) is usually 0[n) which is clearly for large n significantly smaller than 0(n 3 ). However, the 
techniques based on coding/decoding of Expander codes usually do not allow for j3 to be as large as % . 

The main subject of this paper will be the algorithms from the second class. Namely, the second class 
assumes the algorithms that should be designed without having freedom to design the measurement matrix 
A in parallel. If one has no freedom in the choice of the matrix A (instead the matrix A is rather given 
to us) then the recovery problem £[]) becomes NP-hard. The following two algorithms (and their different 
variations) are then of special interest (and certainly have been the subject of an extensive research in recent 
years): 

1. Orthogonal matching pursuit - OMP 

2. Basis matching pursuit - l\-optimiz.ation. 

Under certain probabilistic assumptions on the elements of the matrix A it can be shown (see e.g. [63, 64, 
86, 88]) that if a = 0{(3 log(i)) OMP (or a slightly modified OMP) can recover x in £T|) with complexity 
of recovery 0(n 2 ). On the other hand the so-called stage-wise OMP from [35] recovers x in (Q]) with 
complexity of recovery 0(n log n). 

In this paper we will mostly be interested in the second of the two above mentioned algorithms, i.e. we 
will be mostly interested in the performance of l\ -optimization. (Variations of the standard l\ -optimization 
from e.g. [16, 19,77]) as well as those from [26,43,48-50,76] related to ^-optimization, < q < 1 
are possible as well.) Basic l\ -optimization algorithm finds x in £T|) by solving the following £i-norm 
minimization problem 

min ||x||i 

subject to ^4x = y. (2) 

Quite remarkably, in [15] the authors were able to show that if a and n are given, the matrix A is given 
and satisfies a special property called the restricted isometry property (RIP), then any unknown vector x 
with no more than k = (5n (where is an absolute constant dependent on a and explicitly calculated 
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in [15]) non-zero elements can be recovered by solving ©. As expected, this assumes that y was in fact 
generated by that x and given to us. The case when the available measurements are noisy versions of y is 
also of interest [14, 15,51,92]. Although that case is not of primary interest in the present paper it is worth 
mentioning that the recent popularity of l\ -optimization in compressed sensing is significantly due to its 
robustness with respect to noisy measurements. (Of course, the main reason for its popularity is its ability 
to solve £T|) for a very wide range of matrices A; more on this universality from a statistical point of view 
the interested reader can find in [33].) 

Clearly, having the matrix A satisfy the RIP condition is of critical importance for previous claim to hold 
(more on the importance of the RIP condition can be found in [13]). For several classes of random matrices 
(e.g., matrices with i.i.d. zero mean Gaussian, Bernoulli, or even general Sub-gaussian components) the RIP 
condition is satisfied with overwhelming probability [1,5, 15,74]. (Under overwhelming probability we in 
this paper assume a probability that is no more than a number exponentially decaying in n away from 1.) 
However, it should be noted that the RIP is only a sufficient condition for l\ -optimization to produce the 
solution of (H)). 

Instead of characterizing the mxn matrix A through the RIP condition, in [27,28] the author associates 
certain polytope with the matrix A. Namely, [27, 28] consider polytope obtained by projecting the regular 
n-dimensional cross-polytope using the matrix A. It turns out that a necessary and sufficient condition for 
(fSJ) to produce the solution of (Q~|) is that this polytope associated with the matrix A is fc-neighborly [27-30]. 
Using the results of [2, 10, 60, 73, 90], it is further shown in [28], that if the matrix A is a random mxn 
ortho-projector matrix then with overwhelming probability polytope obtained projecting the standard n- 
dimensional cross-polytope by A is ^-neighborly. The precise relation between m and k in order for this to 
happen is characterized in [27,28] as well. 

It should be noted that one usually considers success of (fSJ) in finding solution of £T|) for any given x. 
It is also of interest to consider success of © in finding solution of (Q~|) for almost any given x. To make a 
distinction between these cases we recall on the following definitions from [28,29, 31]. 

Clearly, for any given constant a < 1 there is a maximum allowable value of the constant (3 such that 
© finds solution of © with overwhelming probability for any x. This maximum allowable value of the 
constant is called the strong threshold (see [28]). We will denote the value of the strong threshold by (3 S . 
Similarly, for any given constant a < 1 one can define the sectional threshold as the maximum allowable 
value of the constant such that © finds the solution of © with overwhelming probability for any x with 
a given fixed location of non-zero components (see [28]). In a similar fashion one can then denote the value 
of the sectional threshold by (3 sec . Finally, for any given constant a < 1 one can define the weak threshold 
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as the maximum allowable value of the constant j3 such that (O finds the solution of (Q} with overwhelming 
probability for any x with a given fixed location of non-zero components and a given fixed combination of 
its elements signs (see [28]). In a similar fashion one can then denote the value of the weak threshold by 
(3 W . In this paper we determine the values of (3 S , (3 sec , j3 w for the entire range of a, i.e. for < a < 1, for a 
specific group of randomly generated matrices A. 

We organize the rest of the paper in the following way. In Section [2] we introduce two key theorems that 
will be the heart of our subsequent analysis. In Section [3] we determine the values of various thresholds in 
the case of general sparse signals x under the assumption that the null-space of the matrix A is uniformly 
distributed in the Grassmanian. Under the same assumption on the statistics of the measurement matrix A 
in Section |4] we determine the values of the weak threshold in a special case of the so-called signed vectors 
x. Finally, in Section [5] we discuss obtained results and possible directions for future work. 

2 Key theorems 

In this section we introduce two useful theorems that will be of key importance in our subsequent analysis. 
First we recall on a null-space characterization of the matrix A that guarantees that the solutions of £[]) and 
(O coincide. The following theorem from [81] provides this characterization (similar characterizations can 
be found in [32,34,42,57,83,95,99]; furthermore, if instead of l\ one, for example, uses an ^-optimization 
(0 < q < 1) in (0 then characterizations similar to the ones from [32, 34, 42, 57, 83, 95, 99] can be derived 
as well [48-50]). 

Theorem 1. (Null-space characterization; General xj Assume that an m x n measurement matrix A is 
given. Let x be a k-sparse vector whose non-zero components can be both positive or negative. Further, 
assume that y = Ax and that w is ann x 1 vector. Let K be any subset of {1,2, ... ,n} such that \K\ = k 
and let Ki denote the i-th element of K. Further, let K = {1,2, ... ,n}\K. Let 1 be a 2 k x k sign matrix. 
Each element of the matrix 1 is either 1 or —1 and there are no two rows that are identical. Let lj be the 
j-th row of the matrix 1. Then (f2]) will produce the solution o/([7]) if 

n—k 

(Vw G R n \Aw = 0) and VK,j - ljW K < ^ |w^.|. (3) 

Remark: The following simplification of the previous theorem is also well-known. Let w £ R n be such 
that Aw = 0. Let |w|(j\ be the i-th smallest magnitude of the elements of w. Setw = (|w|n), |w|( 2 ), . . . , |w|( n )) T . 
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If (Vw|j4w = 0) J27=n-k+i ™i — Y%=i ™i> w here Wj is the i-th element of w, then the solutions of (Q]) 
and (|2]) coincide. While we will make use of this formulation in the following section, the formulation given 
in Theorem Q] will also turn out to be useful for the analysis that will follow in the later sections of the paper. 

Having matrix A such that © holds would be enough for solutions of Q and £0) to coincide. If one 
assumes that m and k are proportional to n (the case of our interest in this paper) then the construction of 
the deterministic matrices A that would satisfy (O is not an easy task (in fact, one may say that it is one 
of the most fundamental open problems in the area of theoretical compressed sensing; more on an equally 
important inverse problem of checking if a given matrix satisfies the condition of Theorem Q] the interested 
reader can find in [25, 54]). However, turning to random matrices significantly simplifies things. As we 
will see later in the paper, the random matrices A that have the null-space uniformly distributed in the 
Grassmanian will turn out to be a very convenient choice. The following phenomenal result from [47] that 
relates to such matrices will be the key ingredient in the analysis that will follow. 

Theorem 2. ( [47] Escape through a mesh) Let S be a subset of the unit Euclidean sphere S 1 ™ -1 in R n . 
Let Y be a random (n — m) -dimensional subspace of R n , distributed uniformly in the Grassmanian with 
respect to the Haar measure. Let 

w(S) = £sup(h T w) (4) 

wes 



where h is a random column vector in R n with i.i.d. M(0, 1) components. Assume that w(S) < [^Jm — ^= 
Then 



P(Yf]S = 0) > l-3.5e n . (5) 

Remark: Gordon's original constant 3.5 was substituted by 2.5 in [75]. Both constants are fine for our 
subsequent analysis. 



3 Probabilistic analysis of the null-space characterizations - general x 

In this section we probabilistically analyze validity of the null-space characterization given in Theorem Q] 
In the first subsection of this section we will show how one can obtain the values of the strong threshold f3 s 
for the entire range < a < 1 based on such an analysis. In the later two subsections we will generalize 
the strong threshold analysis and obtain the values of the weak and the sectional threshold. 
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3.1 Strong threshold 

As masterly noted in [75] Theorem[2]can be used to probabilistically analyze (O (and as we will see later in 
the paper, many of its variants). Namely, let S in (O be 

n n—k 

S s = {weS n ~ 1 \ (6) 

i=n— fe+1 i=l 

where as earlier the notation w is used to denote the vector obtained by sorting the absolute values of the 
elements of w in non-decreasing order. (Here and later in the paper, we assume that k is chosen such that 
there is an < a < 1 such that the solutions of (Q]) and (O coincide.) Let Y be an (n — m) dimensional 
subspace of R n uniformly distributed in Grassmanian. Furthermore, let Y be the null-space of A. Then 
as long as w(S s ) < (^/m — j^=^, Y will miss S s (i.e. © will be satisfied) with probability no smaller 
than the one given in (|5]>. More precisely, if a = ^ is a constant (the case of interest in this paper), n, m 
are large, and w(S s ) is smaller than but proportional to ^fra then P{Y n S s = 0) — ► 1. This in turn is 
equivalent to having 

n n—k 

P(Vw G R dn \Aw = 0, wi < Wi) — >1 

i=n—k+l i=l 

which according to Theorem Q] (or more precisely according to the remark after Theorem [T]) means that the 
solutions of (Q]) and © coincide with probability 1. For any given value of a G (0,1) a threshold value of 
(5 can be then determined as a maximum (5 such that w(S s ) < (^/m — j^=^j ■ That maximum will be 
exactly the value of the strong threshold f3 s . If one is only concerned with finding a possible value for (3 S it 
is easy to note that instead of computing w(S s ) it is sufficient to find its an upper bound. However, as we 
will see later in the paper, to determine as good values of (5 S as possible, the upper bound on w(S s ) should 
be as tight as possible. The main contribution of this work will be a fairly precise estimate of w(S s )- 

In the following subsections we present a way to get such an estimate. To simplify the exposition we first 
set w(h, S s ) = max wg 5 s (h T w). In order to upper-bound w(S s ) we will first in Subsection 13 . 1 . 1 I determine 
an upper bound B s on w(h, S s ). The expected value with respect to h of such an upper bound will be an 
upper bound on w(S s ). In Subsection |3.1.2| we will compute an upper bound on that expected value, i.e. we 
will compute an upper bound on E{B S ). That quantity will be an upper bound on w(S s ) since according to 
the following E{B S ) is an upper bound on w(S s ) 

w{S s ) = Ew{h, S s ) = £(max(h T w)) < E(B S ). (7) 

wes s 
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3.1.1 Upper-bounding w(h, S s ) 

From the definition of set S s given in © it easily follows that if w is in S s then any vector obtain from w 
by changing the signs to any subset of its elements is also in S s . The signs of w can therefore be chosen so 
that they match the signs of the corresponding elements in h. We then easily have 

n n 

w(h, S s ) = max(h T w) = maxy^lhjWjl = max |hJ|wj|. (8) 
wes s wes s wes s f—' 

Let |h|(j) be the i-th smallest magnitude of elements of h. Set h = ((hl^), |h|/ 2 ), • • • , |h|(n)) T - If w € S s 
then a vector obtained by permuting the elements of w in any possible way is also in S s . Then ([8]) can be 
rewritten as 

n 

w(h, S s ) = maxy^hjlwil (9) 
wes s • 
i=i 

where hj is the i-th element of vector h. We will make use of the following simple lemma. 

Lemma 1. Let w be the solution of the maximization on the right-hand side of ((9]). Then |w n | > |w n _i| > 
|w„_ 2 | > > |wi|. 

Proof. Assume that there is a pair of indexes m,n% such that ni < ni and |w ni | > |w n2 |. However, 
|w ni |h ni + |w n2 |h n2 < |w„ 2 |h ni + 1 w ni |h n2 and w would not be the optimal solution of the maximization 
on the right-hand side of (O. □ 

Let y = (yi 

; Y2j • • • j yn) ^ R n - Then one can simplify (O in the following way 

n 

w(h, S s ) = max h^yj 
yeR™ f-f 
i=i 

subject to yj > 0, < i < n 

n n—k 

i=n— fc+1 i=l 
ii 

Ey?<i- do) 

i=i 

To be in complete agreement with Lemma [Done should add the sorting constraints on the elements of y in 
the optimization problem above. However, it is an easy exercise (similar to the proof of Lemma [T]) to show 
that these constraints would be redundant, i.e. it is easy to show that any solution y to the above optimization 
problem will automatically satisfy y n > y„_i > • • • > yi (of course since we will be interested in upper- 
bounding w(h,S s ) one can even argue that in the maximization problem (O dropping constraints would 
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certainly provide a value no smaller than the optimal one obtainable if the constraints are included). To 
determine an upper bound on w(h, S s ) we will use the method of Lagrange duality. Before deriving the 
Lagrange dual we slightly modify (PTOl) in the following way 

n 

- w(h, S s ) = min -V]hjyj 

i=i 

subject to yi > 0, < i < n 

n n—k 
i=n— fc+1 i=l 



i=i 

To further facilitate writing let z G i? n be a column vector such that Zj = 1, 1 < i < (n — k) and 
Zj = — 1, n — k + 1 < i < n. Further, let A = (Ai, A2, • • • , A n ) T G R n . Following, e.g. [1 1], we can write 
the dual of the optimization problem (TTTb and its optimal value w up (h, S s ) as 

- w up (h, S s ) = max min -h T y + 7||y||l - 7 + ^z T y - A T y 

7,f.A y 

subject to v > 0, 7 > 

Aj > 0,0 <i<n. (12) 



One can then transform the objective function in the following way 

tu q \ ■ 11 r- A + h-i/z 2 ||A + h-i/z[|| 
- w up h,5 s = max min ^7y — — 2 ~7 z 

j,v,X y 2,^7 47 

subject to v > 0, 7 > 

Ai>0,0<i<n. (13) 

After trivially solving the inner minimization in (fT3T ) we obtain 

w up (h, b s ) = mm 7 H 

7,^,A 47 

subject to z/ > 0, 7 > 

Aj>0,0<i<n. (14) 

Minimization over 7 is straightforward and one easily obtains that 7 = ^ A+h 2 ~' yz ^ 2 is optimal. Plugging this 
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value of 7 back in the objective function of the optimization problem (fl4l) one obtains 



w up (h,S s ) = mm ||A + h — uz\\ 2 
subject to v > 

Xi>0,0<i<n. (15) 

By duality, — w up (h, S s ) < — w(h, S s ) which easily implies w(h, S s ) < w up (h, S s ). Therefore w up (h, S s ) 
is an upper bound on w(h, S s ). (In fact one can easily show that the strong duality holds and that w(h, S s ) = 
w U p(h, S s ); however, as explained earlier, for our analysis showing that w up (h, S s ) is an upper bound on 
w(h, S s ) is sufficient.) Along the same lines, one can easily spot that any feasible values v and A in (fT31) 
will provide a valid upper bound on w up (h, S s ) and hence a valid upper bound on w(h, S s ). In what follows 
we will in fact determine the optimal values for u and A. However, since it is not necessary for our analysis 
we will not put too much effort into proving that these values are optimal. As we have stated earlier, for our 
analysis it will be enough to show that the values for v and A that we will obtain are feasible in ([TBI . 

To facilitate the exposition in what follows instead of dealing with the objective function given in (fT3T > 
we will be dealing with its squared value. Hence, we set /(h, v, A) = ||A + h — vz\\\. Now, let A = 
(Ai, A 2 ,... , A c ,0,0, ... ,0) T , Ai > A 2 > ••• > A c > where c < (n - k) is a crucial parameter that 
will be determined later. The optimization over v in (fT3T > is then seemingly straightforward. Setting the 
derivative of /(h, v, A) with respect to v to zero we have 

dl|A + h-i/zl|i _ Q 
dv 

^ -2(A + h) T z + 2||z||^ = 
(A + h) r z 

^ v = ~ n ii9 — • (!6) 

hill 

If (A + h) T z > then v = ^rrrre Z * s indeed the optimal in (fT5l) . For the time being let us assume that 

Il z ll2 

A, h, c are such that v = ^ X t^\ z > 0. For v = ^ z we have 

I z ll2 H Z ll2 

(A ^f Z ^ ^ = ||(A + hf (I - ^)\\l = (A + hf (I - ^)(A + h). (17) 

\\z\\2 Z Z Z 1 Z 

Simplifying (fT71) further we obtain 

/(h , A) =± % + 2 £ a* + £ *? - - Ek*>_ _ «k#!), (18) 



z 9 t — ' . — , : — ' n n n 

lz i=i i=i i=i 
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To determine good values for A we proceed by setting the derivatives of /(h, ^t^) 2 z , A) with respect to 

||z|| 2 

Ai, 1 < i < c to zero 

* % y A) - 2A, + 2h f - 2 <£kM _ 2 ffljO _ p. (19) 
aAj n n 

Summing the above derivatives over i and equalling with zero we obtain 



i=l ' i=l i=l 



From (1201 ) one then easily finds 



^ ) = #)_^ (21) 



n — c n — c 
1=1 



Plugging the value for Yli=i Ai obtained in (1211 in ( fT9l ) we have 

A , (h T z) - | (Etl A,) _ (h T z) ^ | c(h^z) ELlhi 



n n re n(n — c) n — c 

and finally 

A, = g±^Jj-i,, 1<<<c 
n — c 

Ai = 0, c + 1 < i < re. (22) 
Combining ([Tol and (1211 we have 

_ (A + h)^z _ h^z + Eti A,. _ h^z + £g) - _ (hT z) _ E c =i fi. 



n n n — c 



(23) 



From (1221 we then have as expected 

v = Xi + hi,l < i < c. (24) 



As long as we can find a c such that Aj , 1 < i < c given in (1221 are non-negative z/ will be non-negative as 
well and v and A will therefore be feasible in (fT5l) . This in turn implies 



w(h,S s ) < y/f(h,v,\) (25) 
where f(h, u, A) is computed for the values of A and v given in (1221 and (1241) . respectively. (In fact deter- 
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mining the largest c such that Aj , 1 < i < c given in (l22l are non-negative will insure that \J /(h, v, A) = 
u;(h, 5 S ); however, as already stated earlier, this fact is not of any special importance for our analysis). 

Let us now assume that c is fixed such that A and v are as given in (l22l) and (l24l ). Then combining (PT8T ). 
(|2TT ). and ([24]) we have 

/(h, ^fi, A) = £ hf +2^ h.-2 ± h* + cu*-2v ± h,+]T hf ~ (ELl A i + hTz)2 • (26) 



i=l 



n 



Combining (|2Tb and (1231 ) we obtain 



Further, combining (1261 ) and (1271 ) we find 



(27) 



(A + hfz 

A h > IMI2 ' A ) 



[nv ) 
n 



8=1 i=l 
n 

1=1 i=l '' ' 



(28) 



Finally, combining (1251 ) and (1281) we have 



w(h,5 s ) < 



((h^-ELih,) 2 



i=l 



«=i 



n — c 



i=c+l 



n — c 



Clearly, as long as (h T z) > there will be a c < n — k (it is possible that c = 0) such that quantity on the 
most right hand side of d29l ) is an upper bound on w(h, S s ). 

To facilitate the exposition in the following subsection we will make the upper bound given in d29l ) 
slightly more pessimistic in the following lemma. 

Lemma 2. Let h £ R n be a vector with i.i.d. zero-mean unit variance gaussian components. Further let 
|h|(j), 1 < i < n, be the i-th smallest magnitude of elements ofh. Set h = (|h|n), |h|( 2 ), • • • , |h|( n )) T 
and w(h, S s ) = max w£ 5 s (h T w) where S s is as defined in (J6]). Let z G R n be a column vector such that 
Zj = 1, 1 < i < (n — k) and Zj = —1, n — k + 1 < i < n. Then 



w(h,S s ) < B s 



(30) 
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where 

n \'E?=ihf if C s (h,c s )<0 

B s 



(31) 

, E^. +1 h?- , "' T t5r- i - ,J </ c,(h,c s) >o' 

Cs(h, c) = ^ h Z ^ n ^' =1 hl — h c arcd c s = 5 s n is a c < n — k such that 

(1 - e)E«jF*) - Eti h,) ! / (l + 6)c \ = Q (32) 



n — c \ n 



F a is the inverse cdf of the random variable \X\ where X is zero-mean, unit variance gaussian random 
variable, e > is an arbitrarily small constant independent ofn. 

Proof. Follows from the previous analysis and (|29l ). □ 
3.1.2 Computing an upper bound on E{B S ) 

In this subsection we will compute an upper bound on E{B S ). As a first step we determine a lower bound 
on P(£ s (h, c s ) > 0). We start by a sequence of obvious inequalities 

F(c, ( h, c.) > o) > p ( «h, c.) > (l^M^SkM + 



n — c s \ n 

>P f((h^)-E& 1 h,) > (i-^((h^)-E& 1 M and F7l ((A±lh_) >il \ 

\ n — c s n-c s V n J J 

^ p / ((h r z)-Etih.) , (l-^((h r z)-Etih,) \ / (l + 6)c a \ ^ 

y n - c s n- c s J V a V n / 

(33) 

The rest of the analysis assumes that n is large so that S s can be assumed to be real (of course, 5 S is a 
proportionality constant independent of n). Using the results from [7] we obtain 

l + e)c s \ . ~ "\ I n f c s (l + e)c s x2 



p rM^l < hc \ < 



exp 



n I " I I o (l+e)c 3 v n n 

1 z n v 

J2 



We will also need the following brilliant result from [20]. Let £(•) : i? n — ► be a Lipschitz function such 
that |£(a) — £(b)| < <r||a — t»|| 2- Let a be a vector comprised of i.i.d. zero-mean, unit variance Gaussian 
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random variables. Then 



P((l - e)^(a) > £(a)) < exp 



2a 2 



(35) 



Let £(h) = (h T z) — YliLi hi- The following lemma estimates <r (for simplicity we assume c s = 0; the 
proof easily extends to the case when c s / 0). 

Lemma 3. Let a, b £ i? n . Let |a|(j), |b|(j) be the i-th smallest magnitudes of a, b, respectively. Set 
a = (l a |(i), |a| (2) , . • • , |a|(„)) andh= (|b| (1) , |b| (2) , . . . , |b| (n) ). Then 



n—k 



n—k 



|£(a)-£(b)| = |E^- E a,-^b l+ ^ b,|<^ 

i=l n—k+l 



i=l n—k+l 



^| ai -b 4 | 2 = v^||a-b|| 2 . (36) 



i=i 



Proof. The following sequence of inequalities/equalities is easy to establish 



n—k n n—k n n—k n—k 

E a * ~ ^ ai - ^ bi + ^2 bi| < | J^(a< -bi)| + | ^ (a— b, ; 

i=l i=n— k+1 i=l i=n—k+l i=l i=n—k+l 



n—k 



n—k 



< 



\aj - bj\ + y~] |aj - bi| < ^ |&j - bj| < \/n 

i=n—k+l 



i=l 



i=l 



i=l 




v Ei a *i 2 + Ei b *i 2 - 2 E a * b * 
\| i=i 



j=i 



i=l 



< Vn 



^|ai| 2 + ^|b i | 2 -2^a i bi = \fn^ 



i=i 



i=i 



i=l 



\ E i ai ~ h{ 

\ i=i 



(37) 



The last inequality follows since the components of a and b are positive and sorted in the same non- 
decreasing order. Connecting beginning and end in (I37T ) establishes (l36l >. □ 



For £(h) = (h T z) — X^i=i hi the previous lemma then gives a < y/n (in fact a < ^Jn — c s ). If n is 
large and S s is a constant independent of n, one can following [78] (as we will see later in the paper) show 
that E((h T z) — Y^iL\ hi) = V'a n where ip s is independent of n as well (ip s is of course dependent on and 
S s ). Hence choosing £(h) = (h T z) — Ym=i hi m d35l > we obtain 



((h T z) - Ei=i hi) ^ (1 - e)£((h r z) - ZZi hi) 



< 



n - c. 



n - c s 



< exp 



2ra 



exp 



e 2 ipfn 



(38) 
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From C33]>, I®, and (ESJ we finally have 



P(C,(h,c) > 0) > i - p ( « £T ') -Sa.fi.) , d-^Kh^-Efc.h,) 



n - c. 



n - c, 



n 



> 1 — exp 



ne 2 5 s 
2(1 + e) 



exp 



e 2 ip 2 n 



(39) 



We now return to computing an upper bound on E{B S ). By the definition of B s we have 



C s (h,c s )<0 ^ 



^ hfp(h)dh + 



8=1 



C s (h,c s )>0 ^ 



i=c s +l 



n - c. 



■p(h)dh (40) 



where p(h) is the joint pdf of the i.i.d. zero-mean unit variance gaussian components of vector h. Since 



the functions yY17=i anc ^ are rotationally invariant and since the region Cs(h, c s ) < takes up the 
same fraction of the surface area of sphere of any radius we have 



„ n 
JCs(h,c a )<0 \J - =1 



53 h? / P (h)dh< 

\ i=1 J(s(h,c s )<0 



n „ 
8=1 J ^ 



(h,c s )<0 



p(h)dh. (41) 



Combining (l39l and (|4TI) we further have 



C s (h,c s )<0 ^ 



J] h?p(h)dh 



i=i 



1=1 



exp 



ne 2 8 s 
'2(1 + e) 



+ exp 



e 2 tp 2 n 



(42) 



It also easily follows 



i=c s +l 



n - c s 



£7. 



1 



i=c s +l 



f.2 ((h^-E-^h,) 2 ^ A £2 (E(hT z )-EE? =1 kY 
n - c. s ~ ^ ' z -* 



i=c s +l 



n - c, 



Finally, the following lemma easily follows by combining (|40l , (|42T ). and (|43l . 



15 



Lemma 4. Assume the setup ofLemma\2\ Let further ij) s = — — —.Then 



E{B S ) < V^(exp|- 



ne 2 5 



2(1+6) 



+ exp 



) + E ± H - (i^mhi . 

(44) 



Proof. Follows from the previous discussion. 



□ 



If n is large the first term in (l44l) goes to zero. Then from (f5]), $J}, and (l44l it easily follows that for a 
fixed a one can determine (3 S as a maximum such that 



a > 



££?= Cs +A 2 (£(h T z) h*) 5 



71 



ran — Co 



(45) 



We recall that k = f3n and z S i? n is a column vector such that Zj = 1, 1 < i < (ra — k) and z, = 
— 1, ra — k + 1 < i < ra. Therefore, in the above equation (5 is hidden in z. It is relatively easy to see that 
problem of finding (3 S for a given fixed a is equivalent to finding minimum a such that (l45l) holds for a fixed 
(3 S . Let /3™ ax be [5 S such that minimum a that satisfies (I43T ) is 1. Our goal is then to determine minimum a 
that satisfies <@5]> for any /3 S G [0, (3™ ax \. 

Therefore in the rest of this subsection we show how the left hand side of d45l) can be computed for a 
randomly chosen fixed j3 s . We do so in two steps: 

1. We first determine c s 

2. We then compute EE ^ s +i h ? - ( g ( hTz )- g E^i h,) 2 wkh ^ found in st L 
From Lemma[2]we have c s = 5 s n is a c such that 



n — c 



n 



n(l-5 s ) 



(46) 



where as in Lemma[2]hi = |h|(i) an d |h|(i) is the i-th smallest magnitude of vector h with i.i.d. zero- 
mean unit variance Gaussian random variables and e > is an arbitrarily small constant. Set 6 S = 1 — S s . 
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Following [8,78] we have 



lim ggkzW>! = r tdFai t). (47) 



n 



We then easily compute F a (1 — 9 S ) in the following way 



e'-dt = l-9 s 



2 

F-\l - 9 S ) = V2erfmv(l - 9 S ) (48) 



o 



where erfinv is the inverse of the standard error function of the normal random variable. We further find 



I" tdF a {t) = r te~*dt = ,/I e "( erfinv (^^)) 2 

Jf-Hi-Os) V vr V vr 

Combining (1471 and (l49l we obtain 

lim E 52"=(i-e s )n+i h i = /2 e -(erfrnv(i-0 s )) 2 _ 

n^oo n V vr 

In a completely analogous way we obtain 

- g E"=(i-/3 3 )n+i h * _ A2 (erfinv(i-/3 3 )) 2 



(49) 



(50) 



lim — wt* = / _ e -iciim n i-^jr (51) 

ra— >oo n V 7T 



Similarly to d48l we easily determine 



2 ^(^j £j /(l + e)tf.n 
' e 2 at - 1 



Jo \ n 

F' 1 (^i^) = ^erfinv (^^) = V2ernnv((l + e)(l - 9 S )). (52) 



Combination of (1461 ). (1501 ). (yTj), and (152] ) gives us the following equation for computing 9 S 

/l e -(erflnv(i-e s )) 2 _ 2 /l e - (erfinv (i-/3 s )) 2 
(1 - e)^ — y — \/2ernnv((l + e)(l - 9 S )) = 0. (53) 

Let 9 S be the solution of (l53l . Then 5 S = 1 — 9 S and c s = J s n = (1 — 9 s )n. This concludes step 1. 
Step 2: 

In this step we compute ' = ^ s+1 ' — — — ~^ n _^ P 1 w i tri c s = (1 ~~ #s)^- Using the results from 
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step 1 we easily find 



lim 

n— >oo 



n(n - c s ) 



2 e -(erfinv(i-e;)) 2 _ 2 /I e -(erfinv(i-/3 s )) 



(54) 



e y^ n h 2 

Effectively, what is left to compute is ' = ^ 3+1 - . Using an approach similar to the one from step 1 and 

following [8, 78] we have 



lim 

n— >oo 



FS^ n h 2 
a Z^i = (l-e s )n+l i 



n 



tdF b (t) 



(55) 



where F b 1 is the inverse cdf of the squared zero-mean unit variance Gaussian random variable. We then 
easily compute i ? fo ~ 1 (l — S ) in the following way 



1 fF^H-Os) e -\ 



2tt Jo 



Vt 



■dt = 1 



Ff x (l - 9 S ) = 2(erfinv(l - s )f. 



(56) 



We further find 



poo / />oo 

La • m(t) = ^ 



2^ ./F-i(i_e s ) 



i 2 

Vie~~dt 



27T + 2- 



exp 



2tt(1-0 s ) 
(57) 



Combining (1551 ) and (1571 ) we obtain 



( 



lim 

n— >oo 



2vr + 2- 



\ 



exp<; 



2tt(1 - Os) 



J 



2(erfinv(l - 6 S )) 2 ^_ 

2vr + 2^ 5 ^- v^tt(1 - a ) I . (58) 

(erfinv(i-e s )) 2 v s/ 1 



We summarize the results from this section in the following theorem. 

Theorem 3. ( Strong threshold) Let Abe anmxn measurement matrix in ([7]) with the null-space uniformly 
distributed in the Grassmanian. Let the unknown x in be k-sparse. Let k, m, n be large and let a = — 
and P. s = — be constants independent of m and n. Let erfinv be the inverse of the standard error function 
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associated with zero-mean unit variance Gaussian random variable. Further, let e > be an arbitrarily 
small constant and Q s , ( f3 s < 9 S < 1 ) be the solution of 



(1-6) 



2 e -(erfinv(i-e s )) 2 _ 2 /][ e -(er/jnv(l-/3 s )) 



V2erfim>{{! + e)(l - 6 B )) = 0. (59) 



If a and (3 s further satisfy 



a > 



2(erfinv(l - 6 S )) 2 
,(erfinv{i-6)y v 



2 e -(erfinv(\-e s )f _ 2 JI e -(erfinv(i-i3 s )) 



(60) 



f/jen the solutions of (0) ant/ (O coincide with overwhelming probability. 

Proof. Follows from the previous discussion combining ©, ©, (|30l . (|44i (l45l> . d53l> . (l54l ). and dSHJ. □ 

The results for the strong threshold obtained from the above theorem as well as the best currently known 
ones from [27,28] are presented on Figure|2] As can be seen, the threshold results obtained from the previous 
analysis are comparable to those from [27, 28] in a large portion of the range for a. For the values of a that 
are close to 1 the threshold values from Theorem[3]are slightly better than those from [27,28]. For a — > 1 
we have j3 m .24 which matches the value obtained in [37] and is in fact optimal. 

Strong threshold, ^-optimization 




Figure 2: Strong threshold, i\ -optimization 
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3.2 Weak threshold 

In this subsection we determine the weak threshold j3 w . Before proceeding further we quickly recall on 
the definition of the weak threshold. Namely, for a given a, j3 w is the maximum value of j3 such that the 
solutions of (Q~|) and (|2]) coincide for any given /3n-sparse x with a fixed location of nonzero components and 
a fixed combination of signs of its elements. Since the analysis will clearly be irrelevant with respect to what 
particular location and what particular combination of signs of nonzero elements are chosen, we can for the 
simplicity of the exposition and without loss of generality assume that the components xi, x 2 , . . . , x n _^ of 
x are equal to zero and the components x„_fc +1 , x„_jt +2 , . . . , x n of x are smaller than or equal to zero. 
Under this assumption we have the following corollary of Theorem [TJ 

Corollary 1. (Nonzero part o/x has fixed signs and location) Assume that an m x n measurement matrix 
A is given. Let x be a k-sparse vector whose nonzero components are negative. Also let xi = X2 = • • • = 
x n -fe = 0. Further, assume that y = Ax and that w is an n x 1 vector. Then (0 will produce the solution 

oftDif 

n n—k 

(VwG« n |Aw = 0) w, ; <^|wi|. (61) 

i=n— i=l 

Following the procedure of Subsection 13. II we set S w 

n n—k 

S w = {weS n - 1 \ Yl w *<El w ^ < 62 ) 

i=n— k+1 i=l 

and 

w(S w ) = E sup (h T w) (63) 

wes,,, 

where as earlier h is a random column vector in R n with i.i.d. M(0, 1) components and S*™ -1 is the unit 
n-dimensional sphere. As in Subsection 13. II our goal will be to compute an upper bound on w(S w ) and 
then equal that upper bound to (^Jm — 4^7^- In the following subsections we present a way to get such 
an upper bound. As earlier, to simplify the exposition we again set w(h, S w ) = max wg s„ (h T w). In order 
to upper-bound w(S w ) we will first in Subsection 13.2. 1 1 determine an upper bound B w on w(h, S w ). The 
expected value with respect to h of such an upper bound will be an upper bound on w(S w ). In Subsection 
13.2.21 we will compute an upper bound on that expected value, i.e. we will compute an upper bound on 
E(B W ). That quantity will be an upper bound on w(S w ) since according to the following E(B W ) is an 
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upper bound on w(S u 



w(S w ) = Ew(h, S w ) = E( max (h T w)) < E{B W ). 

WES,,, 



3.2.1 Upper-bounding w(h, S n 
As in © we have 



n—k 



n—k 



i=l 



i=n—k+l 



max ( 



i=l 



(64) 



w(h,S w ) = max (h T w) = max() IhjwJ + hjWj) = max() IhjllwJ + h. 



iWi 



i=n— k+1 



(65) 

Let h 1:(n _ fe ) = (hi, h2, . . . , h n _fc) T . Further, let now |h||^ _fc ^ be the i-th smallest magnitude of elements 
of h 1:(n _ fe) . Set 



u _ nhi (n_fc) ihi {n_A:) 

n - U n l(l) > l n l( 2 ) > 



I , I (ra— fc) , , v, \T 

l n l( n _fc)) n n-fc+l, n n _fc + 2, . . . , n ra J . 



(66) 



Then one can simplify (1651 ) in the following way 



w(h,S w ) = max h y, 

subject to yi > 0, < i < (n — k) 

n n—k 

E yi > E ^ 

i=n—k+l i=l 
n 

i=i 



(67) 



One can then proceed in a fashion similar to the one from Subsection 13. 1. 1 1 and compute an upper bound 
based on duality. The only differences are that we now have h instead of h and the positive components 
of y are only those with indexes less than or equal to (n — k). After repeating literally every step of the 
derivation from Subsection [37TTT] one obtains the following analogue to the equation (1291 ) 



w(h,S w ) < 



((h T z)-n=ih,) 2 



1=1 



i=l 



n — c 



\ 



E 



((h T z)-n = ih,) 2 



i=c+l 



n — c 



(68) 



where hj is the i-th element of h and c < (n — k) is such that ((h T z) — Ylt=i h») — 0- Clearly, as long as 
(h T z) > there will be a c (it is possible that c = 0) such that quantity on the most right hand side of I 
is an upper bound on w(h, S w ). 
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Using (I68T ) we then establish the following analogue to Lemma[2] 

Lemma 5. Let h E R n be a vector with i.i.d. zero-mean unit variance gaussian components. Further let 
h be as defined in f |66| ) and w(h, S w ) = max w6 5 TO (h T w) where S w is as defined in f \62\) . Let z 6 R n be a 
column vector such that Zj = 1, 1 < i < (n — k) and z^ = — 1, n — k + 1 < i < n. Then 

w(h,S w )<B w (69) 



where 



YS-lH if <*(h,c„)<0 



Cwft 1 , c) = — Z ' > n ^,'" 1 — — h c ant/ c w = 5 w n is a c < n — k such that 



(70) 



(l-e)£((h r z)-£tih*) F -l/ + _ ( , 



n — c a \n(l — (3 W ) 

F^ 1 ^) is the inverse cdfofthe random variable \X\ where X is zero-mean, unit variance gaussian random 
variable, e > is an arbitrarily small constant independent ofn. 

Proof. Follows directly from the derivation before Lemma|2]by replacing h by h and by noting that we now 
have n(l — (3 W ) sorted magnitudes instead of n. □ 

3.2.2 Computing an upper bound on E(B W ) 

Following step-by-step the derivation of Lemma|4](with a trivial adjustment in computing Lipschitz constant 
a) we can establish the weak threshold analogue to it. 

Lemma 6. Assume the setup ofLemma\5\ Let further ip w = — n =1 ' -Then 



E(BJ <^ |CT p<-^} + e* P {-^}) + 



^ 1 n — c 



E 

i=c w +l 

(72) 



Proof. Follows directly from the derivation before Lemma|4] □ 
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As in (|45T ). if n is large, for a fixed a one can determine j3 w as a maximum j3 such that 

^fS^.Wgjg-g^Bjg. (73) 

n n(n — c„,) 

As earlier k = [3n and z G i? n is a column vector such that Zi = 1, 1 < i < (n— fc) and Zj = —1, n—k+1 < 
% < n. Also, as in Subsection 13. 1.21 is again hidden in z. It is not difficult to see that problem of finding 
(5 W for a given fixed a is equivalent to finding minimum a such that (l73l holds for a fixed (3 W . Let f3™ ax be 
/3 W such that minimum a that satisfies d73l is 1. Analogously to what was done in Subsection 13. 1.21 we will 
determine minimum a that satisfies (l73l for any (5 W G [0, P™ ax ]. 

Therefore in the rest of this subsection we show how the left hand side of (l73l can be computed for a 
randomly chosen fixed (3 W . We, as in as in Subsection 13. 1.21 do so in two steps: 

1 . We first determine c w 

2. We then compute 2 - < =°™+ 1 i - z > ^^i n >> w i t h Cw found in step 1. 

71 71) 71 — Cw ) 

Step 1: 

From Lemma [5] we have c such that 

(1 " t)E((E?=f Wn h, " E^n-fcn+l B*) " E -=1 £,) F _, ( (1 + £ ) C = q 



n-c a V"(l-A«) 

^ ^ F ° Ui^j=° (74) 

where we recall hj, 1 < i < (n — P w n), is the z-th smallest magnitude of vector h 1: ( n _^ n ). We also 
recall that h 1: ( n _^ n ) stands for the first (n — /J^n) components of h and hj,n — (9 m n + 1 < % < n, 
are naturally the last (3 w n components of vector h. Also, as always, all components of h are i.i.d. zero- 
mean unit variance Gaussian random variables and e > is an arbitrarily small constant. Then clearly 
Ehi = 0, n — w n + 1 < i < n and we have from (1741 ) 



n — c a \n(l-/3 w ) 

n(i-^) a U(i-^)y 
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Set 8 W = 1 — 5 W . Following [8,78] and in a way completely analogous to (l50l > we obtain 

Hm ^,W h ' = /I e -(ernnv(^)) 2 _ 
ra-*oo n(l — p^J V vr 

As in (l52l) we have 



2 f F » U(i-^) J _£ , /(l + e)^n 
' e 2 dt 1 



Jo \n(l-(3 w ) 

- ^'f 11 ^^)^^"^ 1 ?^)-^™"^^^'- (77) 

\n(l-/3 w )/ W 1 -/^)/ 1-Pu) 

Combination of (|75T ). d76l) . and (1771 ) gives us the following equation for computing 9 W 

/? e -(erfinv(^)) 2 _ 
(1 - e)(l - 2 ^2erfinv((l + e)—^) = 0. (78) 

"■W i PtJ) 



Let to be the solution of d78l) . Then 5 W = I — 6 W and c w = (5 w n = (1 — 9 w )n. This concludes step 1. 
Step 2: 

'„ ^Er =c „ +1 h? ( i?(h^ z ^ 

i(n-c,„) 



In this step we compute gs ^+i h < _ (£(h T z) ^Epi h ' )2 with = (1 - 9 w )n. Using the results 



from step 1 we easily find 



lim i— i '-. 7 = s (79) 

n(n - c^) 



Effectively, what is left to compute is 1 Cw+1 1 . First we note that 



n 

(l-/3 w )n T2 . pv^n v.2 it. v^(l-/3w)n 



— — h Piu- (oU) 

n n n 



Using an approach similar to the one from step 2 of S ubsection 13 . 1 . 2 1 and following [8,78] we have 



lim = / tdFfc(t) (81) 

where as in Subsection 13.1 .21 i 7 ^" 1 is the inverse cdf of squared zero-mean unit variance Gaussian random 
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variable. Following (1561) we then have 



-1/1 "w \ / fi /I "ti)-,>.2 



Tfrsi-^ni))'- <82) 



As in (1571) we also find 



/ 

1 



tdF 6 (t) 
^-1(1^) V2vr 



V 



Combining (18011 . (1811) . (1821 . and (83J we obtain 



2vr + 2 — i ^— - V2vr- 

exp ' v 



(83) 



EE",, / J2(erfinv(f^)) 2 i _ a 

hm = — V2vr + 2-? — : — V2vr- — + fi w . (84) 

n \ (e rfinv(i5fc))2 1 - A. 



n^oo 



We summarize the results from this section in the following theorem. 

Theorem 4. (Weak threshold) Let A be anm x n measurement matrix in ([7]) with the null-space uniformly 
distributed in the Grassmanian. Let the unknown x in (0) be k-sparse. Further, let the location and signs of 
nonzero elements o/x be arbitrarily chosen but fixed. Let k, m, n be large and let a = ^ and f3 w = ^ be 
constants independent of m and n. Let erfinv be the inverse of the standard error function associated with 
zero-mean unit variance Gaussian random variable. Further, let e > be an arbitrarily small constant and 
6w> (Pw < Qui < 1) be the solution of 

p2 - {erfinv (f^-)) 2 

(1 - e)(l - p w )VJL V2erfmv((l + e)— - f ) = 0. (85) 

If a and (3 W further satisfy 

/J -(erfinv (^))^ ' 



(86) 

f/ien ?/ze solutions of (0) arcc? (O coincide with overwhelming probability. 

Proof. Follows from the previous discussion combining ©, ([64K 469K CZ2]), CZS]), dV8]), CZS]), and (HU). □ 
The results for the weak threshold obtained from the above theorem as well as the best currently known 
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ones from [27,28] are presented on Figure[3] As can be seen, the threshold results obtained from the previous 
analysis match those from [27,28]. 

Weak threshold, I -optimization 
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Figure 3: Weak threshold, l\ -optimization 



3.3 Sectional threshold 

In this subsection we determine the sectional threshold j3 sec . Before proceeding further we one more time 
quickly recall on the definition of the sectional threshold. Namely, for a given a, (3 sec is the maximum value 
of P such that the solutions of CD) and © coincide for any given /3n-sparse x with a fixed location of nonzero 
components. Since the analysis will clearly be irrelevant with respect to what particular location of nonzero 
elements is chosen, we can for the simplicity of the exposition and without loss of generality assume that the 
components xi, X2, . . . , x n _^ of x are equal to zero. Under this assumption we have the following corollary 
of Theorem Q] 

Corollary 2 (Nonzero part of x has a fixed location). Assume that anmxn measurement matrix A is given. 
Let x be a k-sparse vector. Also let Xi = X2 = • • • = x n _fe = 0. Further, assume that y = Ax and that w 
is an n x 1 vector. Then ([2]) will produce the solution o/dZ]) 

n n—k 

(VwG« n |Aw = 0) \™i\<^2\ w i\- ( 87 ) 

i=n— fc+1 i=l 
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Following the procedure of Subsection 13.21 we set S sec 

n n~k 

S sec = {weS n - 1 \ M<El Wi l} < 88) 

i=n— fc+1 i=l 

and 

w(S sec ) = E sup (h T w) (89) 



wes. 



sec 



where as earlier h is a random column vector in R n with i.i.d. AA(0, 1) components and S 71 ^ 1 is the unit 
n-dimensional sphere. As in Subsections |3.1| and |3.2| our goal will be to compute an upper bound on w(S sec ) 
and then equal that upper bound to (^/m — -^=\- m the following subsections we present a way to get 
such an upper bound. As earlier, we set w(h, S sec ) = max we s sec (h T w). Following the strategy of previous 
sections in Subsection 13.3. II we determine an upper bound B sec on w(h, S sec ). In Subsection |3.3.2| we will 
compute an upper bound on E(B sec ). That quantity will be an upper bound on w(S sec ) since according to 
the following E(B sec ) is an upper bound on w(S sec ) 

w(S sec ) = Ew(h, S sec ) = E( max (h T w)) < E(B sec ). (90) 

3.3.1 Upper-bounding w(h, S sec ) 
Following (I65T ) we have 



n—k n n—k n 

X " i||Wi 



w(h, S sec ) = max (h r w) = max (S " |hjWj|+ 7^ IhjWjl) = max (V^|hj||wj|+ |h 
weSsec wes sec f— ' . , , wes sec f— f . , , 

i=l i=n— fc+1 i=l i=n— fe+1 

(91) 

As earlier, let h 1: ( n _ fc ) = (hi, h2, . . . , h ra _ K ) T and let |h|/"x be the 2-th smallest magnitude of elements 
of h 1:(n _ fc ). Set 

h = (l h l(i) > l h l(2)~ fc) ' • • • ' l h l(n-fc)' |h„_ fc +i|, |h n _ fc+2 |, . . . , |h n |) T (92) 
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where |hj|, n — k + 1 < i < n, is the absolute value (magnitude) of hj, n — k + 1 < i < n. Then one can 
simplify (I9TI ) in the following way 



w(h, 5 sec ) = max h T y; 



subject to 



yi > 0, < i < n 

n n—k 

E y^E- 

i=n~k+l i=l 
n 

i=i 



(93) 



One can then proceed in the similar fashion as in S ubsection 13 . 1 . 1 1 and compute an upper bound based on 
duality. The only differences is that we now have h instead of h. After repeating literally every step of the 
derivation from S ubsection 13 . 1 . 1 1 one obtains the following analogue to the equation 



w(h, S sec ) < 



\ 



((h T z)-n = ih0 2 



1=1 



i=l 



n — c 



i=c+l 



n — c 



where is the i-th element of h and c < (n — k) is such that ((h T z) — Y^1=\ h») — 0- As earlier, as long 
as (h T z) > there will be a c (it is possible that c = 0) such that quantity on the most right hand side of 
(l94l ) is an upper bound on w(h, S sec ). 

Using (l94l > we then establish the following analogue to Lemmas |2] and [5] 

Lemma 7. Let h G R n be a vector with i.i.d. zero-mean unit variance gaussian components. Further let h 
be as defined in t \92[ and w(h, S sec ) = max we s sec (h T w) where S sec is as defined in t \88[ . Let z G R n be a 
column vector such that Zj = 1, 1 < i < (n — k) and Zj = — 1, n — k + 1 < i < n. Then 



w(h,S sec ) < B 

s 



(95) 



where 



if Csec(h, 

'-sec ) < o 



2^i=c sec +l "i n-c sec y Ssecl«> t-secj U 



£ sec (h, c) = ( h x \ ^i-i h ' _ jj c an< ^ Csec = S sec n is a c < n — k such that 



(96) 



f;M4^i=o, 



n — c 



(97) 
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F a is the inverse cdfofthe random variable \X\ where X is zero-mean, unit variance gaussian random 
variable, e > is an arbitrarily small constant independent ofn. 

Proof. Follows directly from the derivation before Lemma|2] □ 
3.3.2 Computing an upper bound on E(B sec ) 

Following step-by-step the derivation of Lemma|4](with a trivial adjustment in computing Lipschitz constant 
a) we can establish the sectional threshold analogue to it. 

Lemma 8. Assume the setup ofLemma%7\ Let further ip S ec = — « =1 — -Then 



m ec )<^(exp^-^^}+exp{-^^})+ z 



n — c 



sec 



(98) 



Proof. Follows directly from the derivation before LemmalU □ 
As in d45l >. if n is large, for a fixed a one can determine (3 sec as a maximum (3 such that 

a > e Er =Csee+ i n (^(h T z) - e _ 

n n(n - c sec ) 



In the rest of this subsection we show how the left hand side of (1991 can be computed for a randomly chosen 
fixed (3 sec . We again, as earlier, do so in two steps: 



1. We first determine c._ 

£IL Csec+1 h? (E<? 

n(n-c sec ) 



2. We then compute z ^ Csec+1 1 - ( ( \ ^'v 1 ' j with c sec found in step 1. 

77, 77, ( 77 ll c n l 



Step L 

From Lemma|7]we have c sec = 5 sec n is a c such that 

(1 " e)E((Y^-f- n h, - E7=n-p sec n + l hi) - Etr hi) . ( (1 + e)c 



n-c a \n(l-/3 sec ) 



^-^(E^^-E^-P^lM-E^^) (1 + 6)C , ,, 

n-c a v n(l-/? se , N 



(100) 
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where as in Subsection 13.21 |h|& , 1 < i < (w — (3 sec n), is the i-th smallest magnitude of vector 
hi:(„-/3 sec „). Furthermore, h; = |h|^ fc) and clearly h 1:(n _ /3secn ) stands for first (n - /3 sec n) compo- 
nents of h. We also recall that \hi \ , n — (3 sec n + 1 < i < n, are the magnitudes of the last (3 sec n components 
of vector h (these magnitudes of last (3 sec n components of vector h are not sorted). As earlier, all compo- 
nents of h are i.i.d. zero-mean unit variance Gaussian random variables and e > is an arbitrarily small 
constant. Then clearly E\hi\ = y~^, n — j3 sec n + 1 < i < n, and we have from d 100b 

(1 " e)E((Z^- n k ~ £*) " Zkf h.) F _ x ( (1 + 6)0 = o 



n-c a \n(l-B sec ) 

^n-f3 sec n £ pi 



n(l-«S« c ) V»(l-A«c)/ 
Set # sec = 1 — 5 sec . Following the derivation of (1761 ) and (1771 ) we have the following equation for computing 



2 -(erfinv(±efe)) 2 _ Rj3_ 



(1 - e)(l - (3 sec )^ ^ V^-^c _ ^ erfinv((1 + e )L*£~) = 0. (102) 

"sec J- Msec 

Let # sec be the solution of (1 102b - Then 5 sec = 1 — # sec and c sec = 5 sec n = (1 — 6 sec )n. This concludes step 
1. 

Step 2: 

£E™+ih? (E(h^ _ 

i(n-c sec ) 



In this step we compute z ^ Caec+1 1 - ( [ ' _ T 1 ' j with c sec = (1 - (9 sec )n. Using results 



from step 1 we easily find 

((1-3 ),/l p -( erfinv (T5fc)) 2 _a 

lim i— i '- 1 = ± s . (103) 

n-^oo n(n - c sec ) Q sec 

Effectively, what is left to compute is — c ^ ec+1 - ■ However, the same quantity has already been computed 

in d84l ). Hence we have 



,^h 2 / A /2( e rfinv(f^)) 2 i _ w 

\ p (erfmv(i5fc))2 ] ' 

(104) 

We summarize the results from this section in the following theorem. 
Theorem 5. {Sectional threshold) Let A he an m x n measurement matrix in ([7]) with the null-space uni- 
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family distributed in the Grassmanian. Let the unknown x in ([7]) be k-sparse. Further, let the location of 
nonzero elements ofx be arbitrarily chosen but fixed. Let k,m,n be large and let a = ^ and f3 sec = ^ be 
constants independent of 'm and n. Let erfinv be the inverse of the standard error function associated with 
zero-mean unit variance Gaussian random variable. Further, let e > be an arbitrarily small constant and 
&sea ( Psec < ® sec < 1) be the solution of 

R - [erfinv ( i=jfc )) 2 _ R sec 
(1 - e)(l - /3 sec ) — P Vj L ±±^_ _ ^2 erfinv{{l + e )i^£) = 0. (105) 

"sec J- Psec 

If a and (3 sec further satisfy 



n > 1 ^ L I V2vr + 2^- \ M3ec " - V2vr 

\ e (e^w(i5|^)) 2 1-A 



then the solutions of (0) and (0 coincide with overwhelming probability. 




ec 



2 



(106) 



Proo/ Follows from the previous discussion combining ©, (O, d95]), (HU), (|99l), (11021) . (11031 ). and (1104b . 

□ 

The results for the sectional threshold obtained from the above theorem as well as the best currently 
known ones from [27,28] are presented on Figure 01 As can be seen, the threshold results obtained from the 
previous analysis slightly improve on those from [27,28]. 



4 Probabilistic analysis of the null-space characterizations - signed x 

In this section we consider recovery of vectors x with elements known to have certain sign pattern. Without 
loss of generality we assume that it is known that Xj > 0, 1 < i < n. We also again assume that x is 
A;-sparse, i.e. we assume that x has no more than k nonzero elements. To solve (Q~|) for such an x instead of 
© we consider the following optimization problem 

min ||x||i 
subject to j4x = y 

Xi > 0. (107) 

The following theorem from e.g. [81] characterizes the equivalence of (GQ) and (I107I ). 
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Sectional threshold, I -optimization 
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0.9 



Figure 4: Sectional threshold, l\ -optimization 

Theorem 6. (Null-space characterization; Non-negative xj Assume that an m x n measurement matrix A 
is given. Let ~x.be a k-sparse vector whose non-zero components are known to be positive. Further, assume 
that y = Ax. and that w is ann x 1 vector. Let K be any subset of {1, 2, . . . ,n} such that \K\ = k and let 
Ki denote the i-th element of K. Further, let K = {1, 2, . . . , n} \ K. Then < \107i will produce the solution 

ofG)if 

VK and (Vw G R n \Aw = 0, w#. > 0, 1 < i < n - k), 



n—k . 



i= i wk, < E l= i 



w 



Ki- 



(108) 



In the rest of this section we will probabilistically analyze validity of (11081) (or to be more precise, its 
a slight modification). In the first following subsection we will show how one can obtain the values of the 
weak threshold /3+ for the entire range < a < 1 based on such a probabilistic analysis. 

4.1 Weak threshold 

In this subsection we determine the weak threshold Before proceeding further we quickly recall on the 
definition of the weak threshold. The definition of the weak threshold was already introduced in Section 
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13.21 when recovery of general signals (vectors) x was considered. Here, we slightly modify it so that it fits 
the scenario of a priori known sign patterns of elements of x. Namely, for a given a, /3+ is the maximum 
value of j3 such that the solutions of (Q]) and (11071 ) coincide for any given /3n-sparse x with a fixed location 
of nonzero components and a priori known to be comprised of non-negative elements. Since the analysis 
will clearly be irrelevant with respect to what particular location of nonzero elements is chosen, we can for 
the simplicity of the exposition and without loss of generality assume that the components xi, X2, . . . , x n _& 
of x are equal to zero and the components x n _fc +1 , x n _fc +2 , . . . , x n of x are greater than or equal to zero. 
Under this assumption we have the following corollary of Theorem [6l 

Corollary 3. (Nonzero part ofx has fixed a location; The signs of elements ofxa priori known) Assume 
that an m x n measurement matrix A is given. Let x be a k-sparse vector whose nonzero components are 
known to be positive. Also let xi = x 2 = • • • = x n _& = 0. Further, assume that y = Ax and that w is an 
n x 1 vector. Then di07| ) will produce the solution o/([7]) if 

n n—k 

(Vw G R n \Aw = 0,Wj > 0,1 < i < n - k) - ^ Wl <^w;. (109) 

i=n— fc+1 i=l 



Following the procedure of Subsection 13.21 we set 5+ 

n n—k 

5+ = {w g S*™- 1 ! Wi>0, l<i<n-k and - ^ Wi <^Wi} (110) 

i=n— fc+1 i=l 

and 

w(S+) = E sup (h T w) (111) 

where as earlier h is a random column vector in R n with i.i.d. M(0, 1) components and S" 1-1 is the unit 
n-dimensional sphere. As in Subsection 13.11 our goal will be to compute an upper bound on w(S^) and 
then equal that upper bound to (^fm — j^^j- To simplify the exposition we again set w(h,S+) = 
max wg5 + (h T w). We will proceed again as earlier and in Subsection 14.1.11 we will determine an upper 
bound B^ eak on w(h, In Subsection l4.1.2l we will compute an upper bound on E(B^ ]eak ). That quan- 
tity will be an upper bound on w(S+) since according to the following E(B^ jeak ) is an upper bound on 
w(S+) 

w(S+) = Ew(h,S+) = £(max (h T w)) < E(B+ eak ). (112) 

weS£ 
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4.1.1 Upper-bounding w(h, S+) 

In a fashion analogous to ([U) we can write 



n—k 



w(h,S^)= max(h T w) = max hjWj + hjWj). (113) 

1=1 i=n— fc+1 



Let again h 1: ( n _ fc ) = (hi,h2, . . . , h n _fc) T . Further, let fe ^ be the i-th smallest of the elements of 
hwn-fcV Set 



l l:(n-fc)- 

fn-fc) , (n-fc) v,(n-fe) , _ _u . _v, \T 



h+ = (h^h^, . . . , hf^Jj, -h n _, +1 , -h n _ fc+2 , . . . , -h n ) T . (114) 
Then one can simplify d 1 1 31 > in the following way 

w(h, S w ) = max (h + ) T yj 

subject to yj > 0, < i < (n — k) 

n n—k 

i=n—k+l i=l 
n 

I>*<1- (H5) 

i=i 

One can then proceed in the similar fashion as in Subsection 13.1.11 and compute an upper bound based on 
duality. The only differences are that we now have h + instead of h and positive components of y are only 
those with indexes less than or equal to (n — k). After repeating one more time literally every step of the 
derivation from S ubsection 13 . 1 . 1 1 one obtains the following analogue to the equation 



w(h,S+)< 



E|ht| 2 -V|h+'2 (« &+ ) Tz )-n=ih+) 2 



£ |fi +| 2 (((h+) r z)-n=ih+) 2 



n — c 

i=c+l 



\J i=l i=l \ 

(116) 

where h+ is the i-th element of h + and c < (n — k) is such that (((h + ) T z) — Ya=i ^t) — ®- Clearly, as 
long as ((h + ) T z) > there will be a c (it is possible that c = 0) such that quantity on the most right hand 
side of (II 161 ) is an upper bound on w(h, 

Using (11161) we then establish the following analogue to Lemma|2l 

Lemma 9. Let h £ R n be a vector with i.i.d. zero-mean unit variance gaussian components. Further let 
h + be as defined in M14\ and w(h, S+) = max wgS ,+ (h r w) where is as defined in f liiOl ). Let z £ R n 
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be a column vector such that Zj = 1, 1 < i < (n — k) and Zj = —1, n — k + 1 < i < n. Then 



w(h,S+)<B+ eak 



(117) 



where 



B 



weak 



£r=iih, + i 2 



if C(h,c+ )<0 



lyn |h+|2 (((h+rz)-EA e ° fc h+)^ A+f hc + )>0 

Z^;= c + ,+ll n il n _ c + V \ n i C weak> ^ U 



C™(h, c) = — - Z ^_^'~ lh ' h+ anfif c^ eak = <5+n is a c < n — k such that 



(l-6).E(((h+) r z)-ELih+) F _ x / (1+Or 



n — c 



n(l-AT) 



0. 



(118) 



(119) 



F c ij f/je inverse cdf of zero-mean, unit variance Gaussian random variable, e > is an arbitrarily 
small constant independent ofn. 



Proof. Follows directly from the derivation before Lemma|2]replacing h by h + . 



□ 



4.1.2 Computing an upper bound on E(B^ eak ) 

Following step-by-step the derivation of Lemma @] we can establish the following analogue to it. 



Lemma 10. Assume the setup of Lemma® Let further tp v 



E((h+) T z)-E^r fc &+) 



.Then 



£(5+ eafc )<^(expj- 



2(1 + e) 



+ exp 



-n 



+ 



E 



E 



n — c. 



weak 



(120) 



Proof. Follows directly from the derivation before Lemma[4] □ 
As earlier, following (l45l) . if n is large, for a fixed a one can determine /2+ as a maximum /3 such that 

Q > E ^L ak +i '^ +|2 (^((h + ) T z) - gE^r *t)\ (121) 

In the rest of this subsection we show how the left hand side of (11211 ) can be computed for a randomly 
chosen fixed /?+. We will again repeat the two crucial steps: 
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1. We first determine c^ eak 



2. We then compute ^ ^-^ — ' ' + i=1 with c^ eak found in step 1. 



Step 1: 

From Lemma|9]we have c^ eak = S^n is a c such that 

(1 " e)E((E-f in hf - EIU-^+i V) ~ EST K + ) p _ x / (l + e)c = Q 



(122) 

where we recall that now hf , 1 < z < (n — /3+n), is the z-th smallest element (not magnitude) of vector 
^i (n-/3 + n)- Also, we easily have Ehi = 0, n — [3^n + 1 < i < n, and then from (11221 ) 

(1 " ^(0^" h+ - ELn-^n + l & +) " Eg? h+) f (1 + e)c 



n-c \n(l-pS) 

n(l-<5+) c V«(l-^)/ 

Set 0+ = 1 - <J+. Following [8,78] and (EJ we obtain 

lim Mi M^±i = / t( iF c (t). (124) 



We first easily compute F c x ( 1 ) in the following way 



27T Joo 1 /?ui 



^rff^^"^ 2 ^!- 1 )' <125) 



In a similar fashion one then has 



t M+^ = ^ erfiny / (1 +C )(1-^) _ N 
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Using (11251 ) we further find 

r°° n~ f°° t 2 rr -erfmvf^^i-i^ 

/ . tdFJt) = \ — / . te~-dt = a /— e J . (127) 

Combining (1122b . (1 123b . (II 241 ). (11261) . and (11271) we obtain the following equation for finding 0+ 

-erfinv ( 2 -l 



i 

(l-6)(l-/3+) V 



Z7T 



V2erti„v ( 2 d + «>(!;*» _ A _ 0. (,28, 



Let 0+ be the solution of d 128b - Then = 1 — and c^ eafc = <5+n = (1 — #+)n. This concludes step 1. 
Step 2: 

S ^Z" =c + +1 |h+| 2 (E(lii+) T z.)-E s r C "' eak h + ) 2 j_ , 

In this step we compute ^ — — ' ' l — with c^ eak = (1 - 0+)ra. Using 



results from step 1 we easily find 



, ^ l~T - (erfinv (2 J— -i)) 

(i-^)V^ ^ 



2 
2 \ 



]mi (^((h+^-EEgr^) 2 = _v x (129) 



n^oo 



Effectively, what is left to compute is . First we note that 

~r Pw ' 

n n n 

(130) 

Using an approach similar to the one from step 2 of S ubsection 13 . 2 . 2 1 and following [8,78] we have 

Hm Mi = / tdF ^ (131) 

n(l-/3+) Jf~\^4) 

where F^ 1 is the inverse cdf of random variable sign(X)|X| 2 and X is zero-mean unit variance Gaussian 
random variable. Straightforward calculations produce 

J- Pw J- Pru 
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and 



tdFJt) = == 



27T + 2- 



2vr 2- 



1 



V 



cxp 



1-/3B 



(133) 



Combining (fl30l >. dT3Tb . (fT32l . and (fl33l we obtain 



lim 

n— >oo 



^ Z^ = (i_e+) n+ i l"i I 



1 - 8 A 



n 



2V2tt 



2vr + 2 



2(erfmv(2^ 



I)) 2 



(erfinv(2— 



-i)) 2 



1-0+ 
2vr(2- - 1) 



(134) 



We summarize the results from this section in the following theorem. 

Theorem 7. (Weak threshold, a priori known signs ofx) Let Abe an mx n measurement matrix in with 
the null-space uniformly distributed in the Grassmanian. Let the unknown x in ([7]) be k-sparse. Let it be 
known that the nonzero components ofx are positive. Further, let the locations of nonzero elements ofx be 
arbitrarily chosen but fixed. Let k, m, n be large and let a = ^ and /3+ = ^ be constants independent of 
m and n. Let erfinv be the inverse of the standard error function associated with zero-mean unit variance 
Gaussian random variable. Further, let e > be an arbitrarily small constant and ( /3+ < #+ < 1), be 
the solution of 



a > 



i-A 



2V2vr 



(l-e)(l-/3+) 
If a and /?+ further satisfy 

f 

V 



-(erfinv (2 



-I)) 2 



V2erfinv((2 



;i + e)(l-0; 



1-/3. 



+ 



1)) = 0. 



(135) 



2vr + 2 



2(erfinv(2f 



I)) 2 



(erfinv (2^ -1))^ 

-L Pit? 



2tt I 2^—^- - 1 

1-/3* , 



1 1 /T -(«^(2^-i)) a 



(136) 



f/ien ?/ze solutions of ([7]) a«<i rti07D coincide with overwhelming probability. 



Proof. Follows from the previous discussion combining ©, (11121) . dl 17b . dl20| ). (11211 ). dl28l ). dl29l ). and 
(fl34l . □ 
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The results for the weak thresholds obtained from the above theorem in the case of a priori known signs 
of components of x as well as the best currently known ones from [29, 30] are presented on Figure [5] As 
can be seen, the threshold results obtained from the previous analysis match those from [29,30]. 



Weak threshold, ^-optimiz ation, signed x 




Figure 5: Weak threshold, l\ -optimization; signed x 



5 Discussion 

In this paper we considered recovery of sparse signals from a reduced number of linear measurements. We 
provided a theoretical performance analysis of a classical polynomial l\ -optimization algorithm. Under the 
assumption that the measurement matrix A has a basis of the null-space distributed uniformly in the Grass- 
manian, we derived lower bounds on the values of the recoverable strong, weak, and sectional thresholds in 
the so-called linear regime, i.e. in the regime when the recoverable sparsity is proportional to the length of 
the unknown vector. Obtained threshold results are comparable to the best currently known ones. 

The main subject of this paper was recovery of the so-called ideally sparse signals. It is not that difficult 
to see that the presented analysis framework is very general and can be extended to include computations 
of threshold values for recovery of approximately sparse signals as well as those with noisy measurements. 
We should also mention that in this paper we were mostly concerned with the success of l\ -optimization. 
However, the results similar to those presented in this paper can be obtained in the case of £ q -optimization 
(0 < q < 1) as well. All these generalizations will be part of a future work. 
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Carefully following our exposition one can note that the strong threshold results in case of signed vectors 
x are missing. We should mention that the procedure presented in this paper can be repeated for that case 
as well. However, due to a somewhat complicated structure of the set S the analysis of that case becomes a 
bit more tedious and certainly loses on elegance. Nevertheless, we conducted the analysis. However, since 
the final results that we obtained trail those of [29, 30] (except in a very narrow range around a — ► 1) we 
decided not to include them in this paper. 

On the technical side we should mention that our analysis made critical use of an excellent work [47]. On 
the other hand [47] massively relied on phenomenal results [20,68] related to the estimates of the normal tail 
distributions of Lipshitz functions. In a very recent work related to the matrix-rank optimization the authors 
in [70] successfully applied results of [20, 68] directly without relying on the main results from [47]. It will 
certainly be interesting to see what performance guarantees the direct application of the results of [20, 68] 
would produce in the problems considered in this paper. 

At the end we should finally mention a potential universal value of the results presented here. In this 
paper we were mostly concerned with the compressed sensing signal processing applications. However, the 
results presented here may be of independent mathematical interest as well. First, clearly our analysis (as 
almost any other analysis related to compressed sensing) has immediate impact on important mathematical 
problem of solving under-determined systems of linear equations. Second, following the derivations of 
[28,29,32] it is not that difficult to see that our results can be directly applied to determine the neighborliness 
thresholds of projected cross-polytope, regular simplex, and positive orthant as well. 
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